Hello Takeshi,

 

If the empty-block-in-final-frame solution is the preferred one then I think it would be useful to add an example of such an empty block for the cases when there is no compressed data to be sent in the final frame due to the compression stream having been already flushed and byte aligned. DEFLATE implementations like Zlib do not normally produce an empty block encoding when there is no data to flush, so developers will need to have some understanding of the DEFLATE format in order to produce this block artificially. This can result in mistakes. Since it is rare to encounter cases with data aligned in such a manner that there is nothing to send in the final frame, such mistakes might not be detected during development/testing but might leak in production and result in the occasional communication failures. So IMHO adding an example would be useful.

 

Thanks!

 

With best regards,

Arman

 

From: Takeshi Yoshino [mailto:tyoshino@google.com] 
Sent: Tuesday, January 14, 2014 6:05 AM
To: Arman Djusupov
Cc: hybi@ietf.org
Subject: [permessage-deflate] Compressing fragmented data

 

Hi Arman,

 

Let me change the subject.

 

On Thu, Jan 9, 2014 at 11:46 PM, Arman Djusupov <arman@noemax.com> wrote:

Hello Takeshi,

 

In the current draft, the description of the compression process states :

 
   1.  Compress all the octets of the payload of the message using
       DEFLATE.

 

This doesn’t take into account outbound messages of arbitrary size that NEED to be fragmented, for which it is more favorable to use fragmentation and to compress & send fragments until the end of the message is reached.

 

OK. I'll add some text to explain how this works for fragmented messages.

 

 

In such cases the implementation buffers the input from its source and then flushes it out into a compressed fragment, repeating this process until the end of the source data is reached. At that point it flushes any remaining bytes buffered into a final frame, or flushes a final frame of 0 length.

 

 

When the implementation produces compressed fragments, it periodically produces frames with 0x00 0x00 0xFF 0xFF at the end of the frame due to flushing DEFLATE. But because of the following requirements:

 

   2. If the resulting data does not end with an empty DEFLATE block

       with no compression (the "BTYPE" bits is set to 00), append an

       empty DEFLATE block with no compression to the tail end.

 

   3.  Remove 4 octets (that are 0x00 0x00 0xff 0xff) from the tail end.

       After this step, the last octet of the compressed data contains

       (possibly part of) the DEFLATE header bits with the "BTYPE" bits

       set to 00.

 

the implementation must ensure that the message ends empty blocks without a trailing 0x0000FFFF, so the implementation must keep track on what type of block was at the end of the frame that was sent last. If the last frame sent ends with 0x0000FFFF then the implementation cannot remove those 4 bytes from the wire, but it must artificially produce an empty DEFLATE block to send as the final frame. This is only due to the requirement to remove 0x0000FFFF.

 

Yes. It's just two byte long. DEFLATE block header (3bit) + Fixed Huffman code for end of block symbol (7bit) + padding (6bit).

 

 

 

Wouldn’t it be easier to make the removal 0x00 0x00 0xFF 0xFF optional, while at the same time requiring that the receiving side appends a 0x00 0x00 0xFF 0xFF to the final frame in case when its missing?

 

 

I think it complicates decoders. Code which doesn't know DEFLATE decompressor's state cannot check if 0x00 0x00 0xFF 0xFF at the end is really an uncompressed block body or not. It could be something else, e.g. body of compressed block.

 

It is much simpler to just check the final bytes of the message instead of having to remember the state of the previous frame.